NPS-CM-04-026 


,NPS 

PBAtSTXNTlA  PER  SCIENTIAM 


EXCERPT  FROM  THE 

PROCEEDINGS 


OF  THE 

FIRST  ANNUAL  ACQUISITION 
RESEARCH  SYMPOSIUM 


USING  METRICS  TO  MEASURE  CONTRACTOR  PERFORMANCE 

Published:  30  September  2004 
by 

R.  Marshall  Engelbeck 

2nd  Annual  Acquisition  Research  Symposium 
of  the  Naval  Postgraduate  School: 

Charting  a  Course  for  Change: 

Acquisition  Theory  and  Practice  for  a  Transforming  Defense 

May  13,  2004 


Approved  for  public  release,  distribution  unlimited. 

Prepared  for:  Naval  Postgraduate  School,  Monterey,  California  93943 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Report  Documentation  Page 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

13  MAY  2004 


2.  REPORT  TYPE 


3.  DATES  COVERED 

00-00-2004  to  00-00-2004 


4.  TITLE  AND  SUBTITLE 

Using  Metrics  to  Measure  Contractor  Performance 


6.  AUTHOR(S) 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Naval  Postgraduate  School, Graduate  School  of  Business  and  Public 
Policy, 555  Dyer  Road,  Room  332, Monterey, CA, 93943 


5a.  CONTRACT  NUMBER 


5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

5d.  PROIECT  NUMBER 


5e.  TASK  NUMBER 


5f.  WORK  UNIT  NUMBER 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 


11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

1st  Annual  Acquisition  Research  Symposium:  Charting  a  Course  for  Change:  Acquisition  Theory  and 
Practice  for  a  Transforming  Defense,  13  May  2004  in  Monterey,  CA 


14.  ABSTRACT 

see  report 


15.  SUBIECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 


a.  REPORT 

unclassified 


b.  ABSTRACT 

unclassified 


c.  THIS  PAGE 

unclassified 


17.  LIMITATION  OF 

18.  NUMBER 

ABSTRACT 

OF  PAGES 

Same  as 

22 

Report  (SAR) 

19a.  NAME  OF 
RESPONSIBLE  PERSON 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


The  research  presented  at  the  symposium  was  supported  by  the  Acquisition  Chair  of 
the  Graduate  School  of  Business  &  Public  Policy  at  the  Naval  Postgraduate  School. 


To  request  Defense  Acquisition  Research  or  to  become  a  research  sponsor, 
please  contact: 

NPS  Acquisition  Research  Program 
Attn:  James  B.  Greene,  RADM,  USN,  (Ret) 

Acquisition  Chair 

Graduate  School  of  Business  and  Public  Policy 

Naval  Postgraduate  School 

555  Dyer  Road,  Room  332 

Monterey,  CA  93943-5103 

Tel:  (831)656-2092 

Fax:  (831)656-2253 

E-mail:  ibqreene@nps.edu 

Copies  of  the  Acquisition  Sponsored  Research  Reports  may  be  printed  from  our 
website  www.acquisitionresearch.org 

Conference  Website: 
www.researchsymposium.org 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Proceedings  of  the  Annual  Acquisition  Research  Program 

The  following  article  is  taken  as  an  excerpt  from  the  proceedings  of  the  annual 
Acquisition  Research  Program.  This  annual  event  showcases  the  research  projects 
funded  through  the  Acquisition  Research  Program  at  the  Graduate  School  of  Business 
and  Public  Policy  at  the  Naval  Postgraduate  School.  Featuring  keynote  speakers, 
plenary  panels,  multiple  panel  sessions,  a  student  research  poster  show  and  social 
events,  the  Annual  Acquisition  Research  Symposium  offers  a  candid  environment 
where  high-ranking  Department  of  Defense  (DoD)  officials,  industry  officials, 
accomplished  faculty  and  military  students  are  encouraged  to  collaborate  on  finding 
applicable  solutions  to  the  challenges  facing  acquisition  policies  and  processes  within 
the  DoD  today.  By  jointly  and  publicly  questioning  the  norms  of  industry  and  academia, 
the  resulting  research  benefits  from  myriad  perspectives  and  collaborations  which  can 
identify  better  solutions  and  practices  in  acquisition,  contract,  financial,  logistics  and 
program  management. 

For  further  information  regarding  the  Acquisition  Research  Program,  electronic 
copies  of  additional  research,  or  to  learn  more  about  becoming  a  sponsor,  please  visit 
our  program  website  at: 

www.acquistionresearch.org 

For  further  information  on  or  to  register  for  the  next  Acquisition  Research 
Symposium  during  the  third  week  of  May,  please  visit  our  conference  website  at: 

www.researchsymposium.org 


CHARTING  A  COURSE  FOR  CHANGE: 

ACQUISITION  THEORY  AND  PRACTICE  FORA  TRANSFORMING  DEFENSE 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


HAlsUMUflty  HMUt| 


CHARTING  A  COURSE  FORCEiANGE: 

ACQUISITION  THEORY  AND  PRACTICE  FORA  TRANSFORMING  DEFENSE  -  ii  - 


Using  Metrics  to  Measure  Contractor  Performance 


Presenter:  R.  Marshall  Engelbeck,  Lecturer,  Graduate  School  of  Business  &  Public 
Policy,  Naval  Postgraduate  School 


Abstract 

This  Project  examines  the  use  of  metrics  as  a  means  to  measure  contractor 
performance.  It  is  also  designed  to  supplement  the  research  being  done  on  the  “Characteristics 
of  Good  Metrics  for  Performance  Based  Logistics  (PBL)”  by  Professors  Ken  Doerr  and  Ira  Lewis 
along  with  Admiral  Eaton. 

Introduction 

This  Project  examines  the  use  of  metrics  as  a  means  to  measure  contractor 
performance.  It  is  also  designed  to  supplement  the  research  being  done  on  the  “Characteristics 
of  Good  Metrics  for  Performance  Based  Logistics  (PBL)”  by  Professors  Ken  Doerr  and  Ira  Lewis 
along  with  Admiral  Eaton. 

I  can  remember  as  a  second  lieutenant  in  the  Air  Force,  the  only  metric  I  needed  to 
know  was  how  many  aircraft  are  in  commission  and  how  many  you  could  launch.  This  was  a 
lesson  I  learned  very  earlier  in  my  career  when  the  Wing  Commander,  a  very  ambitious  officer, 
kept  reporting  to  higher  headquarters  all  the  aircraft  were  in  commission.  After  about  two  or 
three  weeks  we  had  a  team  visit  the  base.  We  went  to  the  stand-up  in  the  morning  the  team 
leader  asked  the  Wing  Commander,  “how  many  aircraft  were  in  commission?”  When  he 
answered  100%  the  team  leader  told  him  to  launch  them.  Only  about  70%  got  off  the  ground. 
Now  that  I  have  more  experience  and  realize  the  world  is  much  more  complex,  it  is  clear  that 
measuring  performance,  especially  when  it  comes  to  multiple  variables  is  important.  This  is 
critically  important  as  it  relates  to  contractor  performance. 
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Research  Questions 


Research  Questions 

1.  What  is  the  regulatory  foundation  for 
Government  oversight  of  contractor 
performance  after  contract  award? 

2.  Is  there  a  difference  between 
contracting  for  supplies/equipment  and  ^ 
services? 

3.  Is  there  a  difference  between  the 

metrics  used  to  evaluate  contractor 
performance  for  delivering 
supplies/equipment  and  services? _ 

4.  Is  there  a  difference  in  the  type  of 
metrics  valued  by  the  User,  P MO,  PCO 
and  ACO/COR? 


Since  my  research  question  concerns  the  use  of  metrics  to  measure  contractor 
performance  in  an  organization  as  large  as  the  Department  of  Defense  the  first  research 
question  addresses  what  is  the  regulatory  foundation  for  the  use  of  metrics  to  measure 
performance? 

The  second  question:  Is  there  a  difference  between  metrics  applicable  to  contracts  for 
supplies  and  contracts  for  equipment.  This  question  was  selected  because  I  perceive  we 
continue  to  view  the  procurement  practices  as  if  we  were  buying  only  supplies  and  equipment. 
Today,  the  statistics  tell  us  that  60%  of  our  dollars  are  going  to  purchase  services. 

This  brings  us  to  the  third  question:  Are  there  differences  between  the  metrics  used  to 
measure  contractor  performance  when  we  are  buying  supplies  and  equipment  from  the  ones 
from  whom  services  are  purchased? 

These  three  questions  are  addressed  in  phase  one  of  my  research  project  and  represent 
a  majority  of  the  information  that  will  be  provided  this  afternoon. 

Although  it  is  part  of  phase  11,1  will  touch  on  a  fourth  question  in  order  to  illustrate  that 
various  functions  have  their  own  interests.  They  have  different  concerns,  which  are  often 
portrayed  in  the  metrics  they  view  as  most  valuable.  This  is  very  significant  because  it  sends 
mixed  signals  as  to  what  we  value  as  being  important. 


hmsumu  fi»KrivnAt, 
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Regulatory  and  Philosophical  Underpinnings  of  Performance 
Measurement 


Regulatory  and  Philosophical 

Underpinnings  of  Performance 

Measurement 

•  Government  Performance  and  Results  Act 

of  1993 

M  R 

•  National  Performance  Review  (1993) 

E  E 

A  S 

•  Adoption  of  Scorecard  Approach  to 

— s  u 

Measuring  Business  Strategy  (2002) 

U  L 

-  Financial  Perspective 

R  T 

-  Customer  Perspective 

C  Q 

-  Internal  Business  Perspective 

-  Learning  and  Crowing  Perspective 

First  we  have  the  Government  Performance  and  Results  Act  of  1993.  Much  of  the 
literature  on  this  subject  concentrates  on  the  requirement  that  each  of  the  departments  in  the 
Executive  Branch  report  to  Congress  on  program  results  every  two  years.  Frankly  I  didn’t 
realize  the  extent  the  management  philosophy  behind  this  piece  of  legislation  has  affected  the 
way  managers  view  their  management  responsibilities  at  the  grass  roots  level.  As  many  of  you 
know  the  Act  says:  no  longer  is  having  a  program  the  primary  objective,  nor  is  it  the  compliance 
with  process  that  is  most  important.  What  is  paramount  is  the  results  achieved  and  the  fact  you 
also  need  to  measure  the  outcome  in  order  to  evaluate  effectiveness.  I’m  hearing  this 
philosophy  expressed  more  frequently  in  day-to-day  discussions  with  the  students  we  have  in 
class  here  in  Monterey  as  well  as  from  students  from  the  buying  offices  throughout  the  country 
that  participate  in  our  distance  learning  programs. 

The  next  major  event  that  has  affected  the  application  of  metrics  to  the  acquisition 
process  has  been  the  National  Performance  Review,  which  as  you  remember,  also  occurred  in 
1993.  The  Core  Vision  of  that  review  was  that  government  works  for  the  people  and  should  be 
free  from  red  tape  and  useless  rules  no  doubt  facilitated  acquisition  streamlining. 

The  Federal  Acquisition  Streamlining  Act  of  1994  has  been  heralded  as  changing  the 
way  we  buy  goods  and  services.  It  prompted  major  changes  to  the  Federal  Acquisition 
Regulations  (FAR),  which  hit  the  field  in  1995.  As  a  practitioner  I  was  most  appreciative  of  the 
“guiding  principles”  and  “performance  standards”  stated  in  the  preamble  to  the  FAR.1  They  were 


1  FAR  Part  1.102 
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very  significant  because  for  the  first  time  in  my  memory,  and  I  go  back  to  the  Armed  Services 
Procurement  Regulations  (ASPR),  a  procurement  regulation  made  it  clear  the  primary  goal  was 
to  support  the  ultimate  user.  This  revision  to  FAR  also  included  four  results  oriented 
performance  standards  by  which  the  process  can  be  measured.  We  all  owe  a  debt  of  gratitude 
to  Professor  Steve  Kelman,  a  member  of  this  panel,  who  was  head  of  OFPP  at  the  time,  for 
putting  acquisition  in  the  right  perspective. 

Next  on  the  philosophical  side,  we  have  the  “balanced  scorecard”  concept.  Because  the 
federal  government  does  not  operate  in  a  “profit”  environment  some  advocate  use  a  modified 
“balanced  scorecard”  approach  when  applied  to  government  operations.2  “Balanced  scorecard” 
is  a  concept  introduced  by  Robert  S.  Kaplan  and  David  P  Norton  of  Harvard  in  1996.  It  evolved 
from  their  study  of  how  four  corporations  measured  their  performance  and  set  corporate 
strategy.3  This  concept  represents  a  good  guide  for  establishing  a  strategic  measurement 
system.  However,  we  need  to  be  cautious.  We  shouldn’t  just  adopt  what  business  is  doing.  It  is 
important  that  we  take  good  ideas  from  industry  and  adapt  them  to  our  needs. 

You  Get  What  You  Measure! 


You  Get  What  You  Measure!* 

•  You  Should  ...  "State  Objectives  in 
Terms  of  Measurable  Results." 

-  People  Are  Better  Motivated  With 
Measurable  Objectives  Than  Without  Them. 
(Schleh  1961) 

•  "If  You  are  Not  Measuring  You  are  Not 
Managing-You  Are  Just  Along  for  the 
Ride."** 

*  Marshall  W  Meyer,  Rethinking  Performance  Measurement  Cambridge  University  Press, 

UK.  2002,4 

**  Luke  Campbell  and  Brion  Koster.  "Software  Metrics:  Adding  Engineering  Rigor  to  a  Currently 
Ephemeral  Process."  briefing  presented  to  the  McGrummwell  F/A-24  CDR  course,  1995 


As  Dr.  Marshall  Meyer  points  out  “You  get  what  you  measure.” 4  This  means  when  measuring 
performance  it  is  vital  to  consider  what  to  measure  and  the  unintended  consequences  on  what 
is  not  measured.  This  can  lead  to  mixed  signals  as  to  what  is  important. 


2  James  B.  Whittaker,  President’s  Management  Agenda:  A  Balanced  Scorecard  Approach ,  Management 
Concepts,  Vienna,  VA.  2003. 

3  Robert  S.  Kaplan  and  David  P.  Norton,  “Putting  the  balanced  Scorecard  to  Work  /’Harvard  Business 

Review,  September-October  1993. 

4  Marshall  Meyer,  Rethinking  Performance  Measurement,  Cambridge  University  Press,  UK,  2004. 
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We  must  state  our  objectives  in  terms  of  measurable  results.  All  agree  with  the 
comment  made  earlier  that  experience  tells  us,  when  we  say  to  contractors  this  is  what  we  want 
and  what  we  will  measure,  we  do  not  always  say  it  in  a  way  they  can  understand  it.  Some  of 
the  results  we  desire  are  not  quantifiable.  Case  in  point:  How  clean  is  clean?  How  do  you 
measure  clean?  We  want  a  facility  maintained  in  the  cleanest  manner.  How  do  you  define 


that? 


Consider  the  quote  ‘people  are  better  motivated  with  measurable  objectives  than  without 
them’,  made  back  in  1961.  The  lesson  from  this  for  our  students  sitting  in  the  back,  don’t  throw 
away  your  old  textbooks.  This  quote  came  from  the  textbook  titled  “Management  by  Results” 
that  was  popular  in  1961. 5  I  kept  mine  all  these  years.  I  knew  it  would  come  in  handy 
sometime. 

Last,  but  not  least,  is  a  statement  that  moves  metrics  from  a  measurement  to  a 
management  tool.  This  is  a  statement  made  by  Luke  Campbell.  It  was  made  at  a  software 
metrics  conference  in  the  mid-1990’s  where  he  said,  “If  you’re  not  measuring,  you’re  not 
managing.  You’re  just  along  for  the  ride!” 6  That  management  philosophy  says  a  lot. 

Performance  Measurement 

So  what  are  we  talking  about?  Here’s  performance  measurement,  I’ll  let  you  read  the 
definitions  as  they  came  out  of  the  President’s  management  agenda  and  the  scorecard 
approach  published  just  last  year. 7 


Performance  Measurement 

•  Process  of  Assessing  Progress 

Toward  Achieving  Predetermined 

Goals 

•  Includes  Information  on, 

-  Efficiency  Resources  are  Transformed 
into  Quality  Outputs 

-  Extent  to  Which  Customers  are 
Satisfied 

-  Effectiveness  of  Government  Operations 
in  Terms  of  Specific  Contributions  to 
Program  Objectives 


James  B  Whittokcr,  Presidents  Management  Agendas  Balanced  Scorecard  Approach.  Management  Concepts. 
Vienna.  VA.  2003.p  259 


5  Edward  C.  Schleh,  Management  by  Results,  McGraw-Hill,  New  York,  1961. 

6  Luke  Campbell  and  Brian  Koster,  “Software  Metrics:  Adding  Engineering  Rigor  to  a  Currently 
Ephemeral  Process,”  1995. 
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Acquisition  Program  Baseline 


Acquisition  Program  Baseline 

(DoD  5000  Series  Regulations) 

•  User  Performance  Requirements 

•  Schedule  Requirements 

•  Interoperability 

•  Supportability 

-  Cost  of  Ownership 

•  Applicable  Environmental  Requirements 

•  Estimate  of  Total  Program  Cost 

5000  2-ft  dated  4pril  5.  2002.  Paragraph  C  14 


Next,  what  overarching  metrics  do  we  have?  What  are  our  objectives  as  stated  by  from 
OSD?  They  are  cited  in  the  DoD  5000  series  regulations.  These  are  the  performance  factors 
the  program  manager  must  report  on  when  reviewing  program  milestones.  They  are  mandatory 
for  high-value  or  high-priority  programs.  They  include  performance  requirements,  schedule 
requirements,  interoperability,  supportability,  cost-of-ownership,  applicable  environmental 
requirements,  and  estimate  of  total  program  cost.7 8 


7  James  B.  Whittaker,  President’s  Management  Agenda:  A  Balanced  Scorecard  Approach,  Management 

Concepts,  Vienna,  VA.  2003. 

8  Paragraph  C  1.4  of  DoD  5000.2-R  dated  April  5,  2002. 
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ACQUISITION  STANDARDS 


"TO  DELIVER  ON  A  TIMELY  BASIS  THE  BEST 
VALUE  PRODUCT  OR  SERVICE  TO  THE  CUSTOMER 
WHILE  MAINTAINING  PUBLIC  TRUST  AND 
FULFILLING  POLICY  OBJECTIVES." 


1 


SATISFY  CUSTOMER 
IN  COST,  QUALITY, 
AND  TIMELINESS 


MINIMIZE  ADMINISTRATIVE 
OPERATING  COSTS 


FULFILL  PUBLIC 
POLICY  OBJECTIVES 


lONDUCT  BUSINESS  WITH 
INTEGRITY.  FAIRNESS  AND 
OPENNESS 


FAR  1  102 


Moving  into  the  contracting  side,  this  came  out  of  the  Acquisition  Streamlining  Act  FAR  1.102. 
Doctor  Kelman,  thank  you  for  putting  this  in  there,  because  we  in  the  contracting  community 
were  working  so  hard  to  follow  the  regulations,  which  we  sometime  forget  about  keeping  our 
eye  on  the  ball. 

Comment  by  Dr.  Kelman:  Will  you  also  please  thank  Dave  Drabkin  who  played  a  very 
important  role  as  a  civil  servant  and  worked  with  Colleen  Preston  in  OSD. 

Note,  that  performance  standards  flow  upward,  illustrating  that  in  order  to  achieve 
results,  the  acquisition  process  must  successfully  meet  the  standards  established  in  four  areas 
of  performance.  I  would  venture  to  say  each  of  these  areas  are  where  performance  metrics 
would  be  useful. 


M.tSW11*  H»KUv|Ut, 
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Categories  of  Deliverables 


Categories  of  Deliverables 

•  Initial  Delivery  of  Supplies  and  Equipment 

•  Installation  Support  ' 

-  Food  Service 

-  Installation  Maintenance 

•  Support  Weapon  Systems  and  Other 
Mission  Capabilities 

-  Component  Repair 

-  Sustaining  Engineering  Services 

-  Advisory  Services 

-  Research  and  Development 

-  Logistics/Sustainment  Services 

•  Depot  Level  Maintenance 

•  Depot  Level  Support  of  Computer  Systems  * 


As  you  recall  two  of  the  first  three  research  questions  dealt  with  the  three  different  categories  of 
deliverables.  First  is  the  delivery  of  supplies  and  equipment.  This  category  includes  the  High- 
Value  and  High-Priority  systems  that  are  covered  by  the  DoD  5000  series  regulations  we 
discussed  earlier.  This  is  the  area  that  in  the  past  has  received  most  of  our  attention  when  it 
came  to  regulations  and  our  policies.  However,  dollar  wise  we’re  not  buying  as  much  in  this 
area  as  we  did  prior  to  the  end  of  the  cold  war.  Currently  approximately  60%  of  procurement 
dollars  go  to  purchase  services.  This  includes  installation  support,  food  service  and  installation 
maintenance.  Then  we  have  weapons  system  support  and  other  mission  capabilities, 
component  repair,  sustaining  engineering,  advisory  services,  research  and  development, 
logistics/sustainment  services  including  depot  level  maintenance  and  computer  software 
support. 
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Categories  of  Contractor  Deliverables  Spectrum  of  Tangibility 


The  next  research  question  was:  Is  there  is  a  difference  between  contracting  for 
Supplies/equipment  and  buying  services?  A  study  from  the  Journal  of  Supply  Chain 
Management  reported  a  survey  of  about  1400  contracting  personnel  in  the  commercial  world 
and  concluded:  (1)  supplies  and  equipment  are  more  tangible  than  services.  (This  slide 
illustrates  a  range  of  tangibility  as  it  applies  to  the  categories  of  deliverables  purchased  by 
DoD.);  (2)  there  is  a  difference  in  the  degree  of  difficulty  between  writing  requirements  for 
tangible  vs.  intangible  items;  and  (3)  it’s  much  harder  to  write  requirements  documents  for  the 
intangible  items.  9  There  have  been  other  studies  that  reached  the  same  conclusion.  Just 
yesterday  I  was  talking  to  Laura  Baldwin  of  the  RAND  Corporation  and  she  commented  that  she 
continues  to  be  surprised  at  the  problems  the  acquisition  community  is  having  in  stating 
performance-based  requirements.  We’ve  gone  to  defining  “what”  we  want  rather  than  describing 
“how”  we  wanted  it  done.  She  also  said  there  are  more  contracting  issues  that  span  the 
differences  between  the  two  methods  than  she  had  ever  imagined.  I  agree  with  what  Ken  Doerr 
said  previously,  we  still  want  to  measure  the  how. 


9  Larry  R.  Smeltzer  and  Jeffery  A.  Ogden,  “Purchasing  Professional’s  Perceive  Differences  Between 
Purchasing  Materials  and  Purchasing  Services.”  Journal  of  Supply  Chain  Management,  Feb.,  02. 
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Top  Ten  Categories 

CATEGORIES/(PERCENT  AGE) 

RELATIVE 

IMPORTANCE 

-  Quality  and  Process  Control  (24.9) 

l 

-  Continuous  Improvement  (9.2) 

2  — 

-  Facility  Environment  (8  2) 

2 

-  Customer  Relationship  (8.2) 

2 

No 

-  Delivery  (8.1) 

2 

Statistical 

-  Inventory  and  Warehousing  (7.0) 

2 

Difference 

-  Ordering  (5.8) 

2 

-  Financial  Condition  (5  5) 

2  — 

-  Certifications-  ISO  (3.6) 

3 

-  Price  (3.6) 

3 

Penny  M  Simpson  and  Judy  A  Siguaw.  “Measuring  the  Performance  of  Suppliers  An  Analysis  of  Evaluation  Processes," 

Journal  of  Supply  Chain  Management  A  Global  Review  of  Purchasing  and  Supply.  Institute  of  Supply  Management.  February 

2002.  Tobies  III  and  IV,  pages  13  - 14 

To  get  an  idea  what  industry  measures,  this  slide  shows  the  top  ten  supplier 
performance  categories  evaluated  by  over  2,000  randomly  selected  members  of  the  Institute  for 
Supply  Management  (ISM)  and  reported  in  the  Journal  of  Supply  Management.10  Note  that  way 
and  above  the  rest  is  quality  and  the  importance  placed  on  continuous  improvement.  Also  note 
at  the  bottom  is  price,  which  was  evaluated  by  less  than  4%  of  the  respondents.  In  the  middle 
section  are  facility  environment  (9.2%),  customer  relationship  (8.2%),  delivery  (8.1%),  inventory 
and  warehousing  (7%),  ordering  (5.8%),  and  financial  condition  (5.5%).  Notice  there  are  no 
statistical  differences  between  these  categories.  This  tells  us  what  industry  thinks  are  the  most 
important  supplier  performance  indicators.  It  also  says  buyers,  as  a  supplier  or  as  a  producer  of 
a  product,  want  to  have  a  good  supply  base  to  deliver  quality  product. 


10  Penny  M.  Simposn,  Judy  A.  Siguaw  and  Susan  C.  White,  “Measuring  the  Performance  of  Suppliers: 
An  Analysis  of  Evaluation  Processes,  Journal  of  Supply  Chain  Management,  February  2002. 
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How  Commercial  Firms 
Measure  Services 

•  Customer-Provider  Relationship 

-  Call  Center  Metrics 

•  Customer  Satisfaction  Metrics 

-  Overall  Customer  Satisfaction 

•  Operations 

-  Total  Cost  of  Ownership 

-  Occupancy  Cost 

•  Special  Interest 

-  Number  of  Contracts  With  Small  Businesses 

Laura  H  Baldwin.  Frank  Camm.  and  Money  Moore.  Strategic  Sourcing  Measuring  and  Managing  Performance. 
RAND  DB-287-AF.  Santo  Monica.  CA..  2000.  Appendix  B 


Let’s  look  at  services.  In  a  study  published  by  RAND,  Laura  Baldwin  studied  14 
corporations  that  purchase  services  to  determine  what  they  measure.  Customer-provider 
relationship,  like  a  call  center  response,  is  an  important  metric.  The  question  asked  was,  “were 
you  satisfied  with  the  response  you  got  on  the  call?”  This  question  could  be  applied  to  a 
maintenance  technician  after  being  dispatched  to  repair  an  aircraft  on  the  line  or  it  can  also 
apply  to  a  repair  of  a  water  heater  in  a  facility  at  a  Navy  installation. 

Customer  satisfaction  metrics,  overall  customer  satisfaction:  That’s  hard  to  quantify. 
The  only  way  is  through  some  type  of  a  survey.  We  all  get  them  all  the  time;  when  we  stay  in 
hotels,  things  like  that. 

Next  is  Operations:  This  I  found  very  interesting.  Many  companies  measure  the  cost  of 
ownership  of  a  facility.  They  track  the  cost  of  repairs  by  building  or  they  evaluate  its  ownership 
cost  by  the  number  of  occupants. 

Last,  but  not  least,  are  special  interests:  The  number  of  contracts  with  small  business  or 
socioeconomic  programs.  The  message  here  is  the  metrics  are  different  for  services  than  for 
hardware. 
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Conclusions 

1.  GPRA  Confirms  Need  to  Design 
Performance  Metrics  that  Measure 
Program  Results 

2.  Customer  Satisfaction  Should  be  the 
Number  One  Standard  Metric 

3.  There  is  a  Difference  Between  Metrics 
Used  to  Evaluate  Contractor 
Performance  between  Supplies/Equip, 
and  Services? 

4.  There  Should  be  Relatively  Few  Key 
Performance  Indicators  (KPIs) 


In  conclusion;  The  Government  Performance  results  Act  (GPRA)  of  1993  establishes  the 
need  for  government  managers  to  define  desired  program  results  and  to  establish  performance 
metrics  so  that  results  achieved  can  be  measured.  Customer  satisfaction  should  be  the  number 
one  standard  metric. 

There  is  a  difference  between  metrics  used  to  evaluate  contractor  performance  for 
supplies/equipment  and  services. 

There  should  be  relatively  few  key  performance  metrics.  However,  achieving  this  with 
the  number  of  competing  interests  will  be  difficult.  Currently  a  company  working  out  of  New  York 
is  walking  the  halls  of  the  Pentagon  saying  that  it  has  an  IT  program  that  will  measure  contractor 
performance.  It  contains  between  100  and  1,000  factors  that  can  be  measured  and  report 
contractor  performance.  They  advertise  that  their  program  can  be  tailored  to  individual 
contracts.  Their  theme  is,  if  we’re  not  measuring  and  managing  all  the  contract  requirements 
then  the  taxpayer  is  getting  short  changed.  I  would  ask,  if  we  had  to  measure  1 ,000  factors  that 
could  be  measured  in  a  contract  and  managers  feel  there  are  only  10  or  so  of  these  they  feel 
are  important,  then  are  we  paying  for  something  we  really  don’t  need  that  much. 
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5.  Metrics  Selected  Should  be  able  to 
Project  Subsequent  Outcomes  (Results) 

6.  Metrics  Selected  Should  Facilitate 
Continuous  Improvement 

7.  Metrics  Selected  Should  Cross- 
Functional  Boundaries 

8.  Should  Provide  Capability  to  Compare 
Contractors 

9.  Metrics  Selected  Should  be  Used  to 
Award/Penalize  Contractors 


Metrics  selected  should  be  able  to  project  subsequent  outcomes.  They  should  give  you 
the  ability,  by  looking  at  trend  analysis,  where  you  will  be  in  the  future.  Answer  the  question;  will 
this  contractor  be  able  to  support  the  organization’s  mission? 

•  Metrics  selected  should  facilitate  continuous  Improvement. 

•  Metrics  selected  should  also  cross-functional  boundaries.  Our  processes  cross- 
organizational  boundaries  horizontally.  They  are  no  longer  confined  to  functional  stovepipes 
as  we  learned  with  the  introduction  of  the  computer. 

•  The  performance  measurement  system  should  provide  the  capability  to  compare  contractors 
and  should  be  use  to  award/penalize  contractors. 


Final  Thoughts 

Research  results  concludes,  performance  measurement  can  be  used  to  manage 
contractor  performance  after  award.  Private  industry  uses  performance  measurement  primarily 
to  evaluate  the  effectiveness  and  efficiency  of  its  purchasing  department  and  to  guide  contractor 
selection,  However,  the  Government  Performance  and  Results  Act  (GPRA)  of  1996  makes 
measurement  of  contractor  performance  a  very  appropriate  way  to  manage  contracts  in  the 
public  sector. 

GPRA  requires  managers  in  the  executive  branch  to  develop  strategic  plans  as  use 
performance  indicators  to  record  output  and  evaluate  the  outcome  of  each  program.  The  study 
recommends  that  the  Kaplan  and  Norton's  Balanced  Scorecard  method  can  be  adapting  to  the 
contract  in  a  manner  that  would  enable  performance  indicators  to  flow  down  to  major 
contractors.  The  goal  would  be  to  add  joint  accountability  for  results  to  the  buyer-seller 
relationship. 
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Past  performance  is  also  is  an  integral  part  of  the  contract  management  process.  It  is 
primarily  seen  as  a  way  to  mitigate  the  risk  of  selecting  a  contractor  with  a  poor  performance 
track  record  by  reporting  on  contractor  performance  annually.  Examples  of  information  that  must 
be  reported  to  a  centralized  data  base  by  the  contracting  officer  includes,  (a)  A  contractor’s 
record  of  conforming  to  contract  requirements  and  standards  of  good  workmanship,  (b)  A 
contractor’s  record  of  forecasting  and  controlling  cost,  (c)  A  contractor’s  adherence  to  contract 
schedules,  including  the  administrative  aspects  of  performance  contractor’s  history  of 
reasonable  and  cooperative  behavior  and  commitment  to  customer  satisfaction  and,  (d)  A 
contractor’s  business-like  concern  for  the  interests  of  the  customer.  The  study  concludes  that 
both  parties  would  jointly  benefit  managing  these  indicators  concurrent  throughout  the  life  of  the 
contract  rather  than  reporting  after  the  fact. 
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2003  -  2006  Sponsored  Acquisition  Research  Topics 

Acquisition  Management 

■  Software  Requirements  for  OA 

■  Managing  Services  Supply  Chain 

■  Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

■  Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to  Shipyard 
Planning  Processes 

■  Portfolio  Optimization  via  KVA  +  RO 

■  MOSA  Contracting  Implications 

■  Strategy  for  Defense  Acquisition  Research 

■  Spiral  Development 

■  BCA:  Contractor  vs.  Organic  Growth 
Contract  Management 

■  USAF  IT  Commodity  Council 

■  Contractors  in  21st  Century  Combat  Zone 

■  Joint  Contingency  Contracting 

■  Navy  Contract  Writing  Guide 

■  Commodity  Sourcing  Strategies 

■  Past  Performance  in  Source  Selection 

■  USMC  Contingency  Contracting 

■  Transforming  DoD  Contract  Closeout 

■  Model  for  Optimizing  Contingency  Contracting  Planning  and  Execution 
Financial  Management 

■  PPPs  and  Government  Financing 

■  Energy  Saving  Contracts/DoD  Mobile  Assets 

■  Capital  Budgeting  for  DoD 

■  Financing  DoD  Budget  via  PPPs 

■  ROI  of  Information  Warfare  Systems 

■  Acquisitions  via  leasing:  MPS  case 

■  Special  Termination  Liability  in  MDAPs 
Logistics  Management 

■  R-TOC  Aegis  Microwave  Power  Tubes 
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■  Privatization-NOSL/NAWCI 

■  Army  LOG  MOD 

-  PBL  (4) 

■  Contractors  Supporting  Military  Operations 

•  RFID  (4) 

■  Strategic  Sourcing 

■  ASDS  Product  Support  Analysis 

■  Analysis  of  LAV  Depot  Maintenance 

■  Diffusion/Variability  on  Vendor  Performance  Evaluation 

■  Optimizing  CIWS  Life  Cycle  Support  (LCS) 

Program  Management 

■  Building  Collaborative  Capacity 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  Aegis  and  SSDS 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module 
Acquisition 

■  Terminating  Your  Own  Program 

■  Collaborative  IT  Tools  Leveraging  Competence 

A  complete  listing  and  electronic  copies  of  published  research  within  the  Acquisition 
Research  Program  are  available  on  our  website:  www.acquisitionresearch.org 
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